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Abstract 

We argue that flavour mixing, both in the quark and lepton sector, follows 
the minimal mixing pattern, according to which the whole of this mixing is 



basically determined by the physical mass generation for the first family of 
fermions. So, in the chiral symmetry limit when the masses of the lightest 



(it and d) quarks vanish, all the quark mixing angles vanish. This minimal 
pattern is shown to fit extremely well the already established CKM ma- 
^ trix elements and to give fairly distinctive predictions for the as yet poorly 

known ones. Remarkably, together with generically small quark mixing, it 
also leads to large neutrino mixing, provided that neutrino masses appear 
through the ordinary "see-saw" mechanism. It is natural to think that this 
minimal flavour mixing pattern presupposes some underlying family symme- 
try, treating families of quarks and leptons in a special way. Indeed, we have 
found a local chiral SU(3)f family symmetry model which leads, through 
its dominant symmetry breaking vacuum configuration, to a natural realiza- 
tion of the proposed minimal mechanism. It can also naturally generate the 
quark and lepton mass hierarchies. Furthermore spontaneous CP violation is 
possible, leading to a maximal CP violating phase 8 = |, in the framework 
of the MSSM extended by a high-scale SU(3) p chiral family symmetry. 



1 Introduction 



The flavour mixing of quarks and leptons is certainly one of the major prob- 
lems that presently confront particle physics (for a recent review see [Q). 
Many attempts have been made to interpret the pattern of this mixing in 
terms of various family symmetries — discrete or continuous, global or local. 
Among them, the abelian U(l) 0, ||, [§] and/or non-abelian SU (2) || ||, [7j and 
SU (3) || m family symmetries seem the most promising. They provide some 
guidance to the expected hierarchy between the elements of the quark-lepton 
mass matrices and to the presence of texture zeros [|l(J in them, leading to 
relationships between the mass and mixing parameters. In the framework of 
the supersymmetric Standard Model, such a family symmetry should at the 
same time provide an almost uniform mass spectrum for the superpartners, 



with a high degree of flavour conservation |TT[ , that makes its existence even 
more necessary in the SUSY case. 

Despite some progress in understanding the flavour mixing problem, one 
has the uneasy feeling that, in many cases, the problem seems just to be 
transferred from one place to another. The peculiar quark-lepton mass hi- 
erarchy is replaced by a peculiar set of U(l) flavour charges or a peculiar 
hierarchy of Higgs field VEVs in the non-abelian symmetry case. As a result 
there are not so many distinctive and testable generic predictions, strictly 
relating the flavour mixing angles to the quark-lepton masses. 

A commonly accepted framework for discussing the flavour problem is 
based on the picture that, in the absence of flavour mixing, only the parti- 
cles belonging to the third generation t, b and r have non-zero masses. All 
other (physical) masses and the mixing angles then appear as a result of the 
tree-level mixings of families, related to some underlying family symmetry 
breaking. They might be proportional to powers of some small parameter, 
which are determined by the dimensions of the family symmetry allowed oper- 
ators that generate the effective (diagonal and off-diagonal) Yukawa couplings 
for the lighter families in the framework of the (ordinary or supersymmetric) 
Standard Model. 

Recently, a new mechanism of flavour mixing, which we call Lightest Fam- 
ily Mass Generation (LFMG), was proposed |12[ (see the papers in ref. [[13] 
for further discussion). According to LFMG the whole of flavour mixing for 
quarks is basically determined by the mechanism responsible for generating 
the physical masses of the up and down quarks, m u and respectively. So, 
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in the chiral symmetry limit when m u and vanish, all the quark mixing 
angles vanish. This, in fact minimal, flavour mixing model was found to fit 
extremely well the already established CKM matrix elements and give fairly 
distinctive predictions for the yet poorly known ones. 

By its nature, the LFMG mechanism is not dependent on the number 
of quark-lepton families nor on any "vertical" symmetry structure, unifying 
quarks and leptons inside a family as in Grand Unified Theories (GUTs). 
The basic LFMG leads to mass-matrices whose non-zero diagonal and off- 
diagonal elements for the N family case are related as follows: 

M 22 : M 33 : ... : M NN = |M 12 | 2 : \M 23 \ 2 : ... : \M N „± N \ 2 (1) 

This shows clearly that the heavier families (4th, 5th, ...), had they existed, 
would be more and more decoupled from the lighter ones and from each 
other. Indeed, this behaviour can to some extent actually be seen in the 
presently observed CKM matrix elements involving the third family quarks 
t and b. 

The above condition (|1|) is suggestive of some underlying flavour symme- 
try, probably non-abelian rather than abelian, treating families in a special 
way. Indeed, for the observed three-family case, we show that the local family 
SU(3)f symmetry || treating the quark and lepton families as fundamen- 
tal chiral triplets, with an appropriate dominant symmetry breaking vacuum 
configuration, could lead to a natural realization of the LFMG mechanism. 
Other outstanding problems in flavour physics are also addressed within the 
framework of the Minimal Supersymmetric Standard Model (MSSM) appro- 
priately extended by a high-scale SU (3) p chiral family symmetry: the quark 
and lepton mass hierarchies, neutrino masses and oscillations and the possi- 
bility of spontaneous CP violation. 

The paper is organized in the following way. In section 2 we give a gen- 
eral exposition of the LFMG mechanism involving two alternative scenarios. 
In section 3 we present the SU(3)p theory of flavour, treating the quark 
and lepton families as fundamental chiral triplets. We show that there is a 
dominant symmetry breaking vacuum configuration related with the basic 
horizontal Higgs supermultiplets, triplets and sextets of SU(3)f, which leads 
to a natural realization of the LFMG mechanism. While we mainly consider 
the supersymmetric case — just the MSSM — most of our argumentation re- 
mains valid for the Standard Model (and ordinary GUTs) as well. Finally, 
in section 4, we give our summary. 
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2 Minimal Flavour Mixing: the Model 



2.1 Quark Mixing 

The proposed flavour mixing mechanism, driven solely by the generation of 
the lightest family mass, could actually be realized in two generic ways. 

The first basic alternative (I) is when the lightest family mass (m u or 
md) appears as a result of the complex flavour mixing of all three families. It 
"runs along the main diagonal" of the corresponding 3x3 mass matrix M, 
from the basic dominant element M 33 to the element M 2 2 (via a rotation in 
the 2-3 sub-block of M) and then to the primordially texture zero element 
Mn (via a rotation in the 1-2 sub-block). The direct flavour mixing of the 
first and third families of quarks and leptons is supposed to be absent or 
negligibly small in M. 

The second alternative (II), on the contrary, presupposes direct flavour 
mixing of just the first and third families. There is no involvement of the 
second family in the mixing. In this case, the lightest mass appears in the 
primordially texture zero Mu element "walking round the corner" (via a 
rotation in the 1-3 sub-block of the mass matrix M). Certainly, this second 
version of the LFMG mechanism cannot be used for both the up and the down 
quark families simultaneously, since mixing with the second family members 
is a basic part of the CKM phenomenology (Cabibbo mixing, non-zero V c b 
element, CP violation). However, the alternative II could work for the up 
quark family provided that the down quarks follow the alternative I. 

So, there are two possible scenarios for the LFMG mechanism to be con- 
sidered. 

2.1.1 Scenario A: "both of the lightest quark masses m u and m d 
running along the diagonal" 

We propose that both mass matrices for the Dirac fermions — the up quarks 
(U = u, c, t) and the down quarks (D = d, s, b) — in the Standard Model, 
or supersymmetric Standard Model, are Hermitian with three texture zeros 
of the following form: 



Mi 



I Oj 
a* Ai b, L 
\0 b* B t 



) 



i = U, D 



(2) 
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It is, of course, necessary to assume some hierarchy between the elements, 
which we take to be: Bi 3> A{ ~ |6j| ^> |ai|. We derive this hierarchical 
structure from our SU(3)p family symmetry model in section 3.3. The ze- 
ros in the (Mj) n elements correspond to our, and the commonly accepted, 
conjecture that the lightest family masses appear as a direct result of flavour 
mixings. The zeros in (Mj) 13 mean that only minimal "nearest neighbour" 
interactions occur, giving a tridiagonal matrix structure. 

Now our main hypothesis, that the second and third family diagonal mass 
matrix elements are practically the same in the gauge and physical quark- 
lepton bases, means that : 

Bi = (m t , m b ) + 5i A i = (m c , m s ) + 5- (3) 

The components <5j and b\ are supposed to be much less than the masses of 
the particles in the next lightest family, meaning the second and first families 
respectively: 

\Si\ < (m c , m s ) \8[\ < (m u , m d ) (4) 

Since the trace and determinant of the Hermitian matrix Mj gives the sum 
and product of its eigenvalues, it follows that 

5i ~ -{m u , m d ) (5) 

while the 5^ are vanishingly small and can be neglected in further considera- 
tions. 

It may easily be shown that our hypothesis and related equations (|3] - 
|5|) are entirely equivalent to the condition that the diagonal elements (Aj, 
Bi), of the mass matrices Mj, are proportional to the modulus square of the 
off-diagonal elements (a*, bi): 



A 

Bi 



U, D (6) 



For the moment we leave aside the question of deriving this proportion- 
ality condition, Eq. (^), from some underlying theory beyond the Standard 
Model (see section 3) and proceed to calculate expressions for all the ele- 
ments of the matrices Mj and the corresponding CKM quark mixing matrix, 
in terms of the physical masses. 

Using the conservation of the trace, determinant and sum of principal 
minors of the Hermitian matrices Mj under unitary transformations, we are 
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led to a complete determination of the moduli of all their elements. The 
results can be expressed to high accuracy as follows: 



B, = (m t - m u , m b - m d ), 



(7) 

\Q>i\ = {y/m u m c , ^m d m s ) (8) 
|6»| = (V m um, y/m d m b ) (9) 

As to the CKM matrix V, we must first choose a parameterisation appropri- 
ate to our picture of flavour mixing. Among many possible ones, the original 
Euler parameterisation recently advocated |l| is most convenient: 



V 




suCdc - cus D e 
cuc D c + sus D e 
-c D s 




(10) 



(11) 



Here d = sin 6^ d and cu > d = cos 6*^ d parameterise simple rotations 
R X 2 between the first and second families for the up and down quarks 
respectively, while s = sin 9 and c = cos 9 parameterise a rotation between 
the second and third families. This representation of V takes into account 
the observed hierarchical structure of the quark masses and mixing angles. 
The CP violating phase is connected directly to the first and second families 
alone. 

The quark mass matrices Mjj and Md are diagonalised by unitary trans- 
formations which can be written in the form: 



V, 



R 12 R 2^ D 



(12) 

where $ ' D are phase matrices, depending on the phases of the off-diagonal 



elements a,- 



e iai and hi = 1 6, | e 



V 









1 







-iPu 



D 



/ e ia D 


V o 





1 










The CKM matrix is defined by 

v = Vu v D = (**)* K)" K) 



-1 



(13) 



(14) 
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and, after a suitable re-phasing of the quark fields, we can use the represen- 
tation 



-i(j> 



D 



R 



-1 









(15) 



The rotation matrices i? 2 3 D an d R\2 ° f° r our mass matrices are readily 
calculated and the CKM matrix expressed in terms of quark mass ratios 



su 




SD 




(16) 



D*7 



(17) 



m b y m t 

and two phases <p = an — olu and 7 = (3d — Pu- 
ll follows that the Cabibbo mixing is given by the well-known Fritzsch 



formula [14 



W I ~ 



s v - s D e 



m s V m c 



(18) 



which fits the experimental value well, provided that the CP violating phase 
is required to be close to |. So, in the following, we shall assume maximal 
CP violation in the form = |, as is suggested by spontaneous CP violation 
in the framework of SU(3)f family symmetry (see section 3.3). The other 
phase 7 appearing in V c b and V u b 



it; 



cb\ 



IK 



ub 



s v s 



(19) 



can be rather arbitrary, since the contribution to s is relatively small, 

even compared with the uncertainties coming from the light quark masses 
themselves. This leads to our most interesting prediction (with the mass 
ratios calculated at the electroweak scale Hy| ): 



w, 



cb\ 



iT± = 0.038 ±0.007 



(20) 



in good agreement with the current data \V c b\ = 0.039 ± 0.003 |^6| . For 
definiteness we shall assume the phase 7 in s, see Eq. fllTD, to be aligned with 



6 



the CP violating phase 0, again as suggested by SU(3)f family symmetry, 
and take 7 = | . This has the effect of reducing the uncertainty in our 
prediction Eq. (120) from 0.007 to 0.004. Another prediction for the ratio: 



V, 



ub 



cb 



m„ 



(21) 



is quite general for models with "nearest-neighbour" mixing [|TJ. 

2.1.2 Scenario B: "up quark mass m u walking around the corner, 
while down quark mass runs along the diagonal" 

Now the mass matrices for the down quarks and charged leptons Me 
are supposed to have the same form as in Eq. (^|), while the Hermitian mass 
matrix for the up quarks is taken to be: 



Mi 



u 




(22) 



All the elements of Mu can again be readily determined in terms of the 
physical masses as: 



A, 



B, 



m t 



m. 



\cu\ = ^m u m t 



(23) 



The quark mass matrices are diagonalised again by unitary transformations 
as in Eq. (|i~2|) , provided that the matrix Vjj is changed to 



(24) 



where the 1-3 plane rotation of the u and t quarks and the phase matrix $ f 



(depending on the phase of the element cy 
the following way: 



\cjj \ e lctu ) are parameterised in 



#13 



Cl3 





Here s 



13 



-Sl3 

sin ^13 and C13 




u 




(25) 



cos^ 



13- 
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The structure of the CKM matrix now differs from that of Eq. ([14]) as it 
contains the direct 1-3 plane rotation for the up quarks: 



v = vuvh = r& u (V)* (sgy 1 (r*)- 1 



(26) 



although the phases and rotations associated with the down quarks are left 
the same as before. This natural parameterisation is now quite close to the 
standard one [p|. The proper mixing angles and CP violating phase (after 



a suitable re-phasing of the c quark, c 
compact formulae: 



ce 



are given by the simple and 



IK, 



Sl2 




w, 



cb\ 



S23 




ub 



Sl3 




(27) 



and 



5 = au - a D - (3 D (28) 

While the values of \V US \ and |V^,| are practically the same as in scenario 
A and in good agreement with experiment, a new prediction for \Vub\ (not 
depending on the value of the CP violating phase) should allow experiment 
to differentiate between the two scenarios in the near future. 



2.1.3 The CKM Matrix 



V, 



CKM 



(29) 



Our numerical results for both versions of our model, with a maximal CP vio- 
lating phase (see discussion in section 3.3.6), are summarized in the following 
CKM matrix: 

( 0.975(5) 0.222(4) 0.0023(5) A ^ 

0.0036(6) B 
0.222(4) 0.975(8) 0.038(4) 
V 0.0084(18) 0.038(4) 0.999(5) / 

The uncertainties in brackets are largely given by the uncertainties in the 
quark masses. There is clearly a real and testable difference between sce- 
narios A and B given by the value of the V u b element. Agreement with 
the experimental values of the already known CKM matrix elements []T5| 
looks quite impressive. The distinctive predictions for the presently relatively 
poorly known V u b and V t d elements^ should be tested in the near future, by 
experimental data from the B-factories. 

1 Present data from CLEO [Il7] and LEP [M favour scenario B. 
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2.2 Lepton Sector 

The lepton mixing matrix is defined analogously to the CKM matrix: 

U = U u Ue ] (30) 

where the indices v and E stand for v = (z/ e , u^, v T ) and E = (e, /x, r). Our 
model predicts the charged lepton mixing angles in the matrix Ue with high 
accuracy to be: 



8in.9 eft = i — sm6» MT = J — sin0 eT ~O (31) 



\/ m T 



provided that the charged lepton masses follow alternative I, along with the 
down quarks ones, or 



sin0 e/i = O sin6 l AtT = sin# er = J — - (32) 




if they follow alternative II, like the up quarks. We consider only alternative 
I for the charged leptons in our SU(3)p family symmetry model, since the 
alternative II for them would lead in general to the same hierarchy between 
their masses as appears for the up quarks, which is experimentally excluded. 

However, in both cases, these small charged lepton mixing angles will 
not markedly effect atmospheric neutrino oscillations |l!|, which appear to 



require essentially maximal mixing sin 2 29 atm ~ 1. It follows then that the 
large neutrino mixing responsible for atmospheric neutrino oscillations should 
mainly come from the U v matrix associated with the neutrino mass matrix 
in fl30l) . This requires a different mass matrix texture for the neutrinos com- 
pared to the charged fermions; in this connection a number of interesting 
models have been suggested plj , using different mechanisms for neutrino 
mass generation. Remarkably, there appears to be no need in our case for 
some different mechanism to generate the observed mixing pattern of neu- 
trinos: they can get physical masses and mixings via the usual "see-saw" 
mechanism [p0| , using the proposed LFMG mechanism for their primary 



Dirac and Major ana masses. 

Let us consider this pattern in more detail. According to the "see-saw" 
mechanism the effective mass-matrix M v for physical neutrinos has the form 
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M v = -M^M nn Mn (33) 

where Mn is their Dirac mass matrix, while Mnn is the Majorana mass 
matrix of their right-handed components. Therefore, one must first choose 
which of the two alternatives I or II works for the neutrino mass-matrix 
(|33|), particularly for its Dirac and Majorana ingredients, M N and M NN 
respectively. There are in fact four possible cases: (i) alternative I works 
for both the matrices Mn and Mnn ', (ii) alternative I works for Mn, while 
alternative II works for Mnn (hi) alternative I works for Mnn , while 
alternative II works for Mn ; (iv) alternative II works for both the matrices 
Mn and Mnn- It is easy to confirm that the cases (iii) and (iv) are completely 
excluded, since they do not lead to any significant v pu —v T mixing as is required 
by the SuperKamiokande data [[19]. However the first two cases, (i) and (ii), 
happen to be of actual experimental interest when an appropriate hierarchy 
is proposed between the matrix elements in the matrices Mn and Mnn- So, 
we also have two possible scenarios in the lepton sector. 



2.2.1 Scenario A*: "all the lightest lepton Dirac and Majorana 
masses running along the diagonal" 



Let us enlarge first on the case (i) considered recently in ref. p2| . The eigen- 
values of the neutrino Dirac mass matrix Mn are taken to have a hierarchy 
similar to that for the charged leptons (and down quarks) 

M N3 ■ M N2 : M m ~ 1 : y 2 : y\ y^ 0.1 (34) 

and the eigenvalues of the Majorana mass matrix Mnn are taken to have a 
stronger hierarchy 

M NN z : M NN 2 : M Nm ^l:y A :y & (35) 

One then readily determines the general LFMG matrices Mn and Mnn to 
be of the type 

/ ay 3 
Mn^Mns W y 2 ay 2 | (36) 
V ay 2 l 



and 
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M NN3 \ /V y* /3y 3 (37) 



'AW 

where, for both the order-one parameters a and (3 contained in the matrices 
Mjv and M^n, we take an extra condition of the type 

\A-l\<y 2 (A = a,/3) (38) 

according to which they are supposed to be equal to unity with a few percent 
accuracy Substitution in the seesaw formula ( |33"|) generates an effective 
physical neutrino mass matrix M v of the form: 



My 



M 2 f y \ 



'^3 



M 



NN3 



(39) 



y i + (y-y 2 ) 2 ^-(y-y 2 ) 
V o l - (y - y 2 ) l J 

The physical neutrino masses are then given with an accuracy of 0(y) taken 
by: 



1 _ V3sM_l 

- 2 Mjv7v 3 



- G-^ir^-V, (40) 



,1 V / 3 N M 2 m fn . M 2 m 



"2 2 M NN3 M NN3 

Taking the Dirac mass parameter Mn 3 to approximately equal the top quark 
mass m t , the Majorana mass parameter can be determined to be M^^ 3 ~ 
1 ■ 10 15 GeV from the experimentally observed value of Am 2 tm ~ 3 ■ 10~ 3 eV 2 
[|19|j (one may neglect the changes in the lepton masses and mixings coming 
from running from the top mass scale to the MeV scale). 

The mass matrix (|39| ) gives essentially maximal — v T mixing in agree- 
ment with the atmospheric neutrino data ||19|| . Also it is in agreement, while 



marginal, with the known LMA (large mixing angle MSW) fit to the solar 



neutrino experiments f23fl . The standard two flavour atmospheric and solar 
neutrino oscillation parameters are expressed in terms of the leptonic CKM 
matrix U as follows: 

sin 2 20 sun = A\U el \ 2 \U e2 \\ sin 2 2# atm = 4|f/ M3 | 2 (l-|^3| 2 ) (41) 
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assuming the approximate decoupling condition /U e z/ 2 <C 1 (see below). The 
predicted values of these parameters in scenario A* are |]22|: 



sin 2 29, 



atm 



sin 2 20, 



2 

3' 



U. 



1 



e3 



2V2 



Am2 sun 
^ m atm 



y 2 (42) 



to be well compared with the experimentally allowed regions fl9| , [23 
sponding to the LMA solution for the solar neutrino problem 



corre- 



0.82 < sin 2 29 atm < 1, 0.65 < sin 2 29 sun < 1 
\U e3 \ 2 < 0.05, 3 ■ 10~ 3 < f" < 5 ■ 10~ 2 (43) 

^ m atm 

2.2.2 Scenario B*: "the lightest Majorana mass walking around 
the corner, while the lightest Dirac masses run along the 
diagonal" 

We now turn to case (ii), which appears to lead to another case of neutrino 
oscillations where together with essentially maximal mixing of atmospheric 
neutrinos the small mixing angle solution MSW (SMA) naturally appears for 
solar neutrinos. We again take the alternative I form Q3"6D for the Dirac mass 
matrix M^, while for the Majorana mass matrix M^n we take the general 
alternative II form 

/ (3y q \ 
M NN ~M NN A yP (44) 
\Py* 1 J 

with an arbitrary eigenvalue hierarchy of the type 



M NN3 : M NN2 : M Nm ~ 1 : yP : y 2q (45) 

where p and q are positive integers. Again, for the parameters a and j3 
in the mass-matrices Mn and M^n, the extra condition (|38D is assumed. 
The seesaw formula (|33| ) then generates an effective physical neutrino mass 
matrix M v which for any p > 2q — 1, q > 1 has a particularly simple form 
(inessential higher order terms are omitted) 
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M 2 

IVI NN3 



( y 2 y y \ 



y 1 - y P-^+ 2 1 + yP-9- 1 (46) 
\y l + yv-o- 1 1 / 

One can see that the matrix ( pj|) automatically leads to the large (maximal) 
Vfj, — v T mixing and small v e — mixing for any hierarchy in the Majorana 
mass matrix ( |4*4| ) satisfying the condition p > 2q — l, q > 1 mentioned above. 
Taking, for an example, p = 5 and q = 3 one can find for neutrino masses 
the values 



M m y 2 3 ML ML , . . 



L _N3 „.-l 

MnN3 3 2 .\/.v.\:i .1/.V.\:| 



while the following predictions for the two flavour oscillation parameters nat- 
urally appear 



2 1 Am 2 9 

sin 2 2# aim ^l, sin 2 2^ n ^-y 2 , U e3 ^ —y, ~ -y 2 (48) 

y V 2 Lxm atrn lb 



Again taking y « 0.1, our predictions fl4"8|) turn out to be inside of the 
experimentally allowed intervals |Q for the SMA solution for solar neutrino 



oscillations: 



0.82 < sin 2 2# atm <l, 10~ 3 < sin 2 26 sun < 10~ 2 

\U e3 \ 2 < 0.05, 5 ■ 10~ 4 < =■ i un < 9 • 10~ 3 (49) 

Ami 

Note that in contrast to the LMA case (|42|) , one must include now even the 
small contributions stemming from the charged lepton sector (see Eq. (PT|) 
into the solar neutrino oscillations. 

So, one can see that the LFMG mechanism works quite successfully in 
the lepton sector as well as in the quark sector. Remarkably, the same 
mechanism results simultaneously in small quark mixing and large lepton 
mixing (in scenario B* for the second and third families). On the other 
hand, there could be different sources of the large neutrino mixing. An 
attractive alternative method for generating large neutrino mixing within 
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the supersymmetric Standard Model is via -R-parity violating interactions, 
which can give radiatively induced neutrino masses and mixing angles in just 
the area required by the current observational data P3L . 



3 The SU(3) F Theory of Flavour Mixing 
3.1 Motivation 

The choice of a local chiral SU(3)p symmetry || as the underlying flavour 
symmetry beyond the Standard Model is based on the following three "pil- 
lars": 

(i) It provides a natural explanation of the number three of observed 
quark-lepton families, correlated with three species of massless or light (m u < 
Mz/2) neutrinos contributing to the invisible Z boson partial decay width 

0; 

(ii) Its local nature could be expected by analogy with the other funda- 
mental symmetries of the Standard Model, such as weak isospin symmetry 
SU(2)w or colour symmetry SU(3)c- A stronger motivation follows from 
superstrings (at least it is one of the most important model-independent re- 
sults derived from perturbative string theory); there are no global non-abelian 
symmetries in the 4D string models (for a review see |26[ ) ; 



(iii) Any family symmetry should be completely broken at low energies 
in order to conform with reality. This symmetry should be chiral, rather 
than a vectorlike one under which left-handed and right-handed fermions 
have the same group-theoretical features. A vectorlike symmetry would not 
provide any mass protection and would give degenerate rather than hier- 
archical quark and lepton family mass spectra. Interestingly, both known 
examples of local vectorlike symmetries, electromagnetic U(1)em and colour 
SU(3)c, appear to be exact symmetries. Also, in standard GUT models 
fermions and antifermions can lie in the same irreducible representation; so 
a GUT ® SU(3)h model necessarily has a chiral family symmetry. 

So, if one takes these naturality criteria seriously, all the candidates for 
flavour symmetry can be excluded except for local chiral SU (3) p symmetry. 
The chiral U(l) symmetries 0, |3], |J do not satisfy the criterion (i) and 
are in fact applicable to any number of quark-lepton families. Also, the 
chiral SU(2) symmetry can contain, besides two light families treated as 
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its doublets, any number of additional (singlets or new doublets of SU{2)) 
families. All the global non-abelian symmetries are excluded by criterion (ii), 
while the vector like ones are excluded by the last criterion (hi). 

Among the applications of the SU(3)p symmetry, the most interesting 
ones are the description of the quark and lepton masses and mixings in the 
Standard Model and GUTs ||, neutrino masses and oscillations [^7| and 
rare processes ||28|| . Remarkably, the SU(3)p invariant Yukawa coupling are 
always accompanied by an accidental global chiral U(l) symmetry |28|, p9f| , 
which can be identified with the Peccei-Quinn symmetry |30| provided it 
is not explicitly broken in the Higgs sector, thus giving a solution to the 



strong CP problem. In the SUSY context plj , the SU(3)f model leads to a 



special relation between fermion masses and the soft SUSY breaking terms 
at the GUT scale, so that all the dangerous super symmetric flavour-changing 
processes are naturally suppressed ^2 



Another sector of applications is related with a new type of topological 
defect — non-abelian cosmic strings appearing during the spontaneous break- 
ing of the SU(3)f — considered as a possible candidate for the cold dark 
matter in the Universe |3~3|1 . And the last point worthy of note is that the 
local chiral SU(3)f symmetry has been applied to GUTs not only in a di- 
rect product form, such as SU(5) ® SU(3) F §, 50(10) ® SU(3) F [0 or 
E(6) ®SU(3)f j33], but also as a subgroup of the family unified SU(8) GUT 
model |3l 



3.2 The Matter and Higgs Supermultiplets 

In the MSSM extended by the local chiral SU(3)f symmetry the quark and 
lepton superfields are supposed to be SU(3)f chiral triplets, so that their 
left-handed (weak-doublet) components 



U L 
D L 

fu 
d 



N L 



c 
s 



e 



(50) 



T 



L \ / L \ / L/ \ \ / L \ r / l \ / L/ 

are triplets of SU(3)f , while their right-handed (weak-singlet) components 
Kf = [U%,N%\ = [(u,c,t) R ,(N e ,N fl ,N T ) R ] (51) 
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R% = [D%,Eft = [(d,8,b) R ,(e,ii,T) R ] 

are anti-triplets (or vice versa). For completeness we have included the right- 
handed neutrino superfields as well. Here a is a family symmetry index 
(a = 1,2,3), while the index / simply stands for the type of basic fermion, 
quark (/ = q) or lepton (/ = /). 

The matter sector also includes some set of right-handed states, which 
are needed to cancel all the SU (3) p triangle anomalies. While being singlets 
under the Standard Model SU{3)c®SU{2) w ®U{1)y gauge symmetry, these 
states can be chosen as sixteen right-handed triplets r™ (n = 1, .., 16) or some 
other combination of SU(3)f multiplets properly compensating the triangle 
anomalies. All of them receive heavy masses of order the family symmetry 
breaking scale Mp and are practically decoupled from the low-energy particle 
spectrum. In fact, they look like right-handed heavy neutrinos and, as such, 
can contribute to the ordinary light neutrino masses and mixings via the 
"see-saw" mechanism [|20] (see below). 

In addition to the standard weak-doublet scalar supermultiplets H and 
H of the MSSM, the Higgs sector contains the SU(3)p symmetry-breaking 
horizontal scalars. These scalars are anti-triplets rf and sextets X* °f SU(3)p 
(indices i and j number the scalar multiplets, i = 1,2,..., j = 1,2,...), which 
transform under SU(3)p like matter bi-linears so as to give fermion masses 
through the symmetry-allowed general effective Yukawa couplings: 



W Y = LXfHJAi r ^ + Bi, r j^] + (52) 

for the up and down fermions, quarks (/ = q) and leptons (/ = I). Also 
Majorana couplings are included for the right-handed neutrinos, where only 
the symmetrical parts are allowed to appear. The constants A l u ^ d ^ N and 
Buf,df,N (Au q = A\j, A\i = A % N and so on) are the dimensionless Yukawa 
constants associated with the symmetric and anti-symmetric couplings in 
family space respectively (we have here used the anti-symmetric feature of 
triplets in SU{3): V\ai3] = e a/3-yV 1 )- These couplings normally appear via the 
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well-known see-saw type mechanism [||, |8| due to the exchange of a special 
set of heavy (of order the flavour scale Mp) vectorlike fermions or directly 
through the gravitational interactions, being suppressed in the latter case by 



inverse powers of the Planck scale Mpi |36| . One can even make the Yukawa 
couplings (|52|) renormalizable, by introducing additional scalars with both 
electroweak and horizontal indices. The VEVs of the horizontal scalars If 
and x\ taken in general as large as Mp (or Mpi), are supposed to be hi- 
erarchically arranged along the different directions in family space, so as to 
properly imitate the observed quark and lepton mass and mixing hierarchies 
even with coupling constants A l u j- d j N and B l u j d j- N taken of the order 0(1). 
The Majorana couplings of the right-handed neutrinos N 1 ^ do not, of course, 
involve the Weinberg-Salam Higgs field H u and are thus naturally of order 
O(Mp). As was mentioned above, the Yukawa couplings fl52|) admit an ex- 
tra global chiral symmetry U(l) identified with the Peccei-Quinn symmetry^ 
U(1)pq ||30|| . However, in general, one must also consider the Higgs super- 
potential (see below), in order to determine whether there exists global U(l) 
or discrete Zn symmetry(ies) in the model. 

There is a more economic way of generating effective Yukawa couplings, 
by using just horizontal triplets rf a and anti-triplets and imitating sextets 
by the pairing of triplet scalars. Instead of the couplings (p2|) , in this case 
[28|j one has: 

W Y = L a f R p uf H4A%.^ a rj^f (53) 
+T f R%H d [A% ■ r? a rf/« + B df ■ ffarb] + 
+N%Ng[A% N ■ 77^I fe - 



NN 



where all the horizontal "bold" scalar fields rf are taken now to be scaled by 
the SU(3)p symmetry breaking scale Mp, rf a = rj l a /Mp. The hierarchy in 
the corresponding mass matrices (including the up-down hierarchy) depends 
solely on the powers fc„ d N and n l udN of the specially introduced hierarchy- 
making singlet scalar field I (I =1 /Mp) in (0). This description turns out 
to be well fitted to the supersymmetric case where both types of scalar, rf a 
and its SUSY counter-part rf a = e^rfip^, appear automatically. 

2 Even in the non-supersymmetric Standard Model which contains only one Higgs dou- 
blet this symmetry can be successfully implemented || ps|[ , by assigning the U(1)pq 
charges directly to the horizontal scalars rf and x*. 
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The basic Yukawa couplings (0) (or (0)) can lead to various types of 
fermion mass matrices, depending on the vacuum configurations developed 
by the horizontal triplets rf\ anti-triplets fj 1 and sextets \ % involved. Re- 
markably, a texture-zero structure emerges as one of the generic features 
of a typical SU(3)f symmetry breaking pattern. This is controlled by the 
antisymmetric (anti-triplet) VEVs < rf > giving off-diagonal elements in 
the mass matrices, while the symmetric (sextet) ones < x 1 > generate the 
dominant diagonal elements. Therefore, any of the texture-zero models cat- 
alogued in the literature (see |T0f| ) are readily available in the framework of 
the SU(3)f model. 

For example, the original Fritzsch ansatz |14| with the zero mass-matrix 
elements, M\\ = M22 = M13 = 0, while presently excluded by experiment, 
appears from a basic vacuum configuration of the Higgs potential || (or 
superpotential |JT|) with a simple set of horizontal scalars — two anti-triplets 
rj\ a m and £r Q( gi and one sextet X{a/3} developing VEVs along the following 
directions in family space^]: 

V[12] = -V[21] > £[23] = -?[32]> X{33} (54) 

The "improved" Fritzsch ansatz (see [Q) with a non-zero M22 element ap- 
pears, when a second sextet tU{ Q/ 3} is included?], which develops a VEV cc>{22}- 
Unfortunately, this ansatz is not completely predictive of the CKM matrix 
in terms of quark mass ratios (which would require no more than three in- 
dependent parameters for each of the mass matrices). In particular it gives 
no possibility of predicting the V c b element in the CKM matrix. 

As a matter of fact the only ansatz left, which successfully predicts all the 
CKM matrix elements in terms of quark mass ratios (not to mention here 
some eclectic approaches using ad hoc GUT and string arguments, which 
are already largely excluded by experiment) is that suggested by the LFMG 
mechanism. In the next subsection, we show how the LFMG can be derived 
within the SU(3)f model. 

interestingly, the VEVs %33 and V[i2] m the coherent basis correspond to the flavour 
democracy picture |l| for the mass-matrices of quarks and leptons with symmetrical and 
antisymmetrical "pairing forces" respectively, while the VEV £[231 breaks this picture 
appropriately. 

4 Both of these ansatze, original and "improved" , readily appear in a pure triplet real- 
ization ( [53| ) as well, with three and four scalar triplets respectively. 
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3.3 Minimal flavour mixing from family symmetry 
3.3.1 Basic vacuum configurations 

We start with an essential question: how does one get the proportionality 
condition ([]]) underlying the LFMG in the framework of the SU(3)f model? 
In terms of the Yukawa superpotential fl5"2|) , this condition (P means that 
not only the VEVs of the horizontal supermultiplets, sextets x\ a p} an d anti- 
triplets V[ a f3] °f SU(3)p , but also their Yukawa coupling constants A l u ^ N 
and B l udN must be properly arranged. Since the coupling constants are in 
general quite independent even in the framework of SU(3)p, this in turn 
means that the number of horizontal supermultiplets involved in the Yukawa 
couplings should be restricted. At the same time, some of these scalars have 
to develop VEVs along several directions in the family space in order to 
satisfy the proportionality condition (D). 

We take the set of scalar supermultiplets in the model to consist of three 
pairs of triplets and anti-triplets and two pairs of sextets and anti-sextets of 
SU(3) F : 

va(rf), UT), C«(T); 

u {an (uJW) (55) 

In section 3.3.2 we will see that just the triplet scalars, to a large extent, 
determine the masses and mixings of the quarks and leptons, while the sextets 
govern the basic vacuum configuration in family space leading to the LFMG 
picture for flavour mixing. 

We take the Higgs superpotential to have the form 

W = M v r J rJ + M^ + M c CC + M xX x + M ul uuJ + 

+f ■ e a ^Va^ + 7 ■ eafHff ? ? (56) 
+F ■ e^e a '^'x {aa , } x {m oo M + F ■ e aPl e a , ^x {aa ' } X {m ^ {W} 

with masses and M X)U) and coupling constants /, /, F and F. One can 

readily notice that all the trilinear couplings of triplets and sextets in the 
superpotential W are very specific just to the SU{3) symmetry. These cou- 
plings by themselves induce, as we will see later, strictly orthogonal (to each 
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other) VEVs for the triplets and sextets involved. However, as was argued 
above, some of these scalars also have to develop hierarchically small paral- 
lel VEVs in the family space, so as to get the proportionality condition ([]]) 
satisfied. For this purpose we include in the superpotential W the additional 
terms 

AW 1 = a-(^-) r M F fjC (57) 



with a particular coupling of triplets rj and £ which, through their mixing, 
will cause some extra small parallel VEVs for them whose magnitudes will 
be determined by the power r of the VEV of the singlet scalar S. This 
hierarchy- making scalar S also appears in the Yukawa couplings (see below). 
We emphasize here that, besides the SU(3)p symmetry, the superpotential 
W has a global U(l) symmetry. This global U(l) is in substance the cus- 
todial symmetry strictly protecting the form of the superpotential W (|5G|) 
considered 5 . 

Let us come now to a general analysis of the non-trivial supersymmetric 
vacuum configurations of the superpotential W . One can quickly find from 
the vanishing F-terms of the triplet supermultiplets involved, 



M vVa + f ■ e aPl fC + a ■ ( iy ) M F ( a = 0, M v rf + f ■ e^&C = 



P-f , ( S 
\M F/ 

M&+J-e aPj rf*C = 0, Msf + f ■ ^«C 7 = 0, 
M c C 7 + /-e a/37 rf = 0, 
M^ + f-e^Va^ + a- (j^MpfT = 0, (58) 

that they develop VEVs satisfying the equations 

, hT = MiMi, &> = W (jr = M^i (59) 



5 Let us note that this £7(1) may be identified with a superstring- inherited anomalous 
U(1)a gauge symmetry |}7| broken at a high scale through the Fayet-Iliopoulos (FI) D- 
term |Q. One can take just this scale as the family symmetry scale Mp. While for the 
heterotic strings this scale normally lies only a few orders of magnitude below the Planck 
mass Mpi, in the orientifold case, where the FI term appears moduli-dependent p9| , it 
can be made to locate much lower. 



20 



All but one of the scalar products between pairs of triplets vanish 



riaC = £ a rj a = U a = (arr = U a = (60) 

= _JL.(JlV m f m< 

and their non-zero components can be chosen as follows: 

r/ Q = (X,0,0), rj a = (X 1 0,W), 

^ = (0,^0), f = (0,Y,0), (61) 
( a = (z,0,Z), ( a = (z,0,Z). 

The components x, z and z which appear in them are small and just arise 
due to the rj( mixing term in the superpotential W mentioned above. We 
propose here that the VEV of the singlet scalar S responsible for this mixing 
is typically somewhat smaller than the basic VEVs of the triplets (|59|) , so as 
to have some input hierarchy parameter j^- in the model (see below). 
The vanishing of the D-terms 

($, T A $) = 0, (62) 

where all the scalars fields (triplets and sextets) are grouped in the vector $ 
{T A are generators of SU(3)f acting on the reducible representation given by 
the vector $), results in further limitations on the possible supersymmetric 
vacuum configurations. Particularly, for the generator T 4+l5 , equation (|62]) 
leads to an additional non-trivial condition on the VEVs of the triplets T} a (jf) 
and ( a (( a ) _ 

- X*x + Z*z-Z*z = 0, (63) 

When this condition is combined with the basic F-term equations fl58|), the 
values of the small VEV components x, z and z can be expressed in terms 
of the VEV of the scalar singlet S (the bold letter S stands for S/M F ) and 
the ratios of the big VEV components as follows: 



x 



-aS r M F 



X Y Z 
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The VEVs slide by themselves into the valleys determined by the F- and 
D-term conditions ( |58l , |6"2"D conserving supersymmetry. 

Let us turn now to the sextet superfields in the superpotential W (]5F 
From their vanishing F-term equations 



M xX{Q/3} + Fe aia tp Sp u^x {(7p} = 0, M x x {aP} + Fe a ^e^u mX{tJp } = °> 
M u u {ap} + Fe a ^s P X {lS} X {r7p} = 0, M x u7^> + Fe^° = 0, 

(65) 

and the corresponding D-term equation fl62|) , one can confirm that the sextets 
develop strictly orthogonal vacuum configurations of the type: 



X{a/3} = X-diag{l,l,0) a p, x {a/3} = X • diag{l, 1, 0) af3 (66) 
u {a/3} = lu ■ diag(0, 0, l) a0 , uJ {af3} = ZJ ■ diag(0, 0, l) a/3 

In order to show this explicitly, one must first rotate away all the compo- 
nents of the sextet X{a/3}, except for xn an d X22, by the appropriate SU(3)f 
transformations. Then, successively using the F-term equations fl5"5]), one 
can see that the sextet uj{ a f3} and the anti-sextet U^ a ^ inescapably develop 
the VEVs given in equation (|66"D with 

M 2 

uuJ= 7 f (67) 

while the anti-sextet x^ a ^ a l so develops VEVs on the same components, (11) 
and (22), as the sextet X{a/3} does. Furthermore, a relation appears between 
them of the form 

u ^22 M x Mu 
XnX - X22X - -p F V ®) 
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Finally, using the D-term equation flB2|) with generators T 3 and T 8 and as- 
suming that the Xa/3 and x VEVs are large compared with all the other 
scalar VEVs contributing to (|B"2"D, one unavoidably comes to the equalities 

= Ix 11 ! = IX22I = \X 22 \ (69) 

Thus we are led to the VEVs for the scalars Xap and x® of the form 
given in equation (|66| ) with \x\ = These VEVs by themselves break 
the family SU(3)p symmetry to the plane SO(2)p symmetry, acting in the 
family subspace of the first and second families of quarks and leptons. We 
will see later that just this vacuum configuration turns out to be essential for 
generating Yukawa couplings, which follow our minimal mixing pattern for 
quarks and leptons. As mentioned above, we assume the VEVs of the sextets 
Xa/3 and x a>3 are the largest in the model, so that this vacuum configuration 
of the sextets is not disturbed by the VEVs of the triplets. However, it should 
be noted that this assumption in no way influences the hierarchy in masses of 
the quarks and leptons, which is completely determined by the triplet scalar 
VEVs given above. The relative alignment assigned to the VEVs of the 
triplet ( |6lD and the sextet ( |66| ) fields is arranged by introducing a coupling 
between the triplet and sextet fields in the superpotential W of the form 

so as not to disturb the above vacuum configurations. This term also excludes 
an accidental global symmetry U {3) triplets® U(3) sext ets in the superpoten- 
tial W which might, otherwise, induce the appearance of extra goldstones 
(familons) after symmetry breaking. Finally, one has the total symmetry 
SU(3) F ® U(l) for the superpotential W + AW 1 + AW 2 considered. 

3.3.2 Yukawa couplings 

One can write down a plethora of SU(3)p invariant Yukawa couplings. How- 
ever, by the use of the additional protecting symmetry U(l), one can choose 
them in a special form which leads to (at least) one of the four experimen- 
tally allowed scenarios for the minimal flavour mixing of quarks and leptons. 
These four scenarios consist of the following four pairs of combined quark and 
lepton flavour mixings (see section 2): A+A*, A+B*, B+A* and/or B+B*. 
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Since we have all the triplet superfields needed in our chosen set (|55|), the 
most economic choice of Yukawa couplings would be the pure triplet realiza- 
tion of the form ( |53"D discussed in section 3.2. However it does not lead to 
the desired strict proportionality condition ([]]) or (|6|) between mass-matrix 
elements of the basic (alternative I) LFMG. So the pure triplet Yukawa cou- 
plings can only be used for the up quarks in scenario B (with its direct 1-3 
mixing) for quark mixing. The up quark Yukawa couplings then take the 
form[]: 

Wyx = U a L U^ R H u [A m • + A U2 ■ ^ p l k + Bu • £ [a/3] F] (71) 

which contains two symmetrical couplings related with the bi-linears for the 
triplet scalars C a and £ a and one antisymmetric coupling related with the 
anti-triplet £r a/3 i from our scalar set (p5|). We recall here that the "bold" 
horizontal scalar fields £ and I are scaled by the SU(3)f symmetry 

mass scale parameter Mp (which can generally be as large as the Plank mass 
Mpi), so that the coupling constants Ajj,nn and Bjj,nn are all dimensionless 
and assumed to be of order unity. The hierarchy in the corresponding mass 
matrix of the up quarks depends, as in the general case fl53"D , solely on the 
powers k and n of the hierarchy-making singlet scalar field I whose values are 
determined by the U(l) symmetry imposed. According to the triplet VEVs 
(^Tp, the first symmetrical coupling in Wyi giving masses to the heaviest third 
family are not suppressed, while all other couplings (including the direct 1-3 
mixing term) are suppressed by powers of the scalar I. As in the general 
case (|53"D, by introducing additional scalars with electroweak and horizontal 
indices combined one could make the Yukawa couplings ( |7T| ) renormalizable. 

The other possible Yukawa couplings, which can be written with the 
triplets (||), are those having the form 

W Y2 = L a f R^ f H u [A uf ■ r7 M % p] + B uf ■ e^fi}^ + (72) 
+LjRdfHd[A d f ■ rj [aa] rj m + B df ■ e a(3(T ri p S l ]x {rTp} + 
+N%N*[A NN ■ rj [a(7] r) m }x Wp} 

6 We do not consider here the mass matrix of right-handed neutrinos to which the 
scenario B+B* is applicable as well. This will be done later in section 3.3.5. 
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for the up and down fermions, quarks (/ = q) and leptons (/ = I), and right- 
handed neutrinos. They contain only one triplet scalar rj a (and its SUSY 
counterpart rj^ = e^rf) developing VEVs ( pH ) along both the first and 
third directions at the same time. They also include one of the sextets 
X^ a/3 \ so as to be properly arranged in the family space. We remark that the 
couplings Wyi Q7ID , like the general Yukawa couplings (|53D, use horizontal 
triplets for their symmetric parts and anti-triplets for the antisymmetric 
parts. However the couplings Wy2 (|72]), on the contrary, prefer to use anti- 
triplets for the symmetric parts and triplets for the antisymmetric parts. 
Just those couplings collected in Wy2 turn out to correspond, as we will see 
later, to the minimal mixing of quarks and leptons with the proportionality 
condition ([I]) between their mass matrix elements. Note that the couplings in 
Wy2 include their own hierarchy making singlet scalar field S. This scalar S 
determines (see Eq.(^)) the small components in the VEVs of the triplets rj 
(rj) and ( (Q and thereby underlie the basic hierarchy between the diagonal 
elements M22 and M33 in the quark and lepton mass matrices (see later). 
Thus, it looks natural that the hierarchy in their off-diagonal elements is 
also dependent solely on the power (/) of the same scalar field S and not 
on the other scalar I introduced in the couplings Wyi (0). Here again 
all the horizontal scalar fields 77, 77, x an d S are scaled by the SU(3)p 
symmetry mass scale parameter Mp, so that the coupling constants A u f^f,NN 
and B u f^f are all dimensionless and assumed to be of order unity. According 
to the U(l) symmetry, this hierarchy should be the same for all the fermions 
involved in Wy2- The symmetric couplings containing the bi-linears of the r\ 
scalar give masses to the fermions of the heaviest third family and have no 
suppression, apart from the contributions coming from the small components 
in its VEV configuration (|6lD. For simplicity, we also propose that the up- 
down hierarchy in the Yukawa couplings ( |72|) is entirely given by the ratio of 
the VEVs of the ordinary Higgs doublets H u and of the MSSM, tan (3 = 
v u /vd, thus considering the large tan/3 case. One can, of course, include 
instead extra powers of the singlet scalar fields S in the Yukawa couplings of 
the down fermions to generate the up-down mass hierarchy. 

The Yukawa couplings ([72]), when taken for all the quarks and leptons in- 
volved, correspond in fact to the combined scenario A+A* for flavour mixing. 
In this case there is only one (natural) hierarchy parameter ys = for all 
the fermion mass-matrices. However, in the combined scenario B+B* where 
the masses and mixings of the up quarks are determined by the Yukawa cou- 
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plings Wyi (f7TD , while those of the down quarks and leptons are determined 
by the Yukawa couplings Wy 2 (|72|) , generally two independent hierarchy pa- 
rameters yi = ^jgj- and ys = ^jf^ appear in the model. In general, one 
can hardly expect that the different Yukawa superpotentials, Wyi and Wy 2 , 
should lead to a similar hierarchy pattern for the corresponding fermions. 
We shall assume (see section 3.3.4) that the hierarchy making scalars I and 
S develop VEVs which are close in value, leading to close values of the hi- 
erarchy parameters yi and ys- However they must have quite different U(l) 
charges, in order to provide the required power-like hierarchy in the quark 
and lepton mass matrices. 

The couplings Wyi (|71~1) and Wy 2 (|72|) are the only Yukawa couplings 
which give masses to the quarks and leptons. Other possible SU(3)p cou- 
plings containing the same fermion and scalar superfields are, in fact, dis- 
allowed by the extra U(l) symmetry mentioned above (see section 3.3.4 for 
more details). 



3.3.3 Mass-matrices of quarks and leptons 

We show now that, from the four presently (experimentally) allowed LFMG 
scenarios for the combined flavour mixing of quarks and leptons, the SU(3) f 
theory admits only one; namely the B+B* scenario which, fortunately, seems 
to be the most preferable with regard to the experimental situation in quark 
mixing (the present value of the V uh element, see section 2.1). 

Towards this end let us first consider the Yukawa couplings Wy 2 (0) 
to understand why they, while being good for all the other fermions, cannot 
work for the up quarks and right-handed neutrinos. Substituting all the VEV 
values of the horizontal triplets (|BTD and sextets as well as those of the 
ordinary MSSM doublets H u and H d , into the Yukawa couplings Wy 2 ([72]), 
one obtains the following mass matrices for the quarks and leptons: 



M 



uf 



M, 



df 



rrr 



m" 



A uf x 2 


( A df x 2 


V A df xX 








A uf x 2 


B u fX.S 


—B u f~X.S l 


A uf X 2 





A df xX 




B d fX.S 


—B d fX.S l 


A df X 2 



(73) 



(74) 
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M; 



NN 



m 



NN 



A NN x 2 

A NN x 2 

A NN xX 



A NN xX 




(75) 



Here the mass parameters mJJ, and m° NN include all the other VEV factors 
appearing with the Yukawa couplings in Wy2 '■ 



rxr 



-<H u >x, m° d =<H d >x, m° NN = M F x (76) 

(the bold letters stand everywhere for the properly scaled VEVs, for example 
X =X/Mp etc). Now, after diagonalization of the 1-3 blocks in the mass 
matrices M u f, and M^n, one immediately comes to the LFMG form (0) 
for the mass-matrices of all the fermions involved (specifying those for the 
up and down quarks, neutrinos, charged leptons and right-handed neutrinos, 
respectively): 



Mi 



u 



M 



D 



M, 



N 



M, 



NN 



m NN 



( ° 

v° 

( 

v° 

(o 

\o 
(o 

/ 

A NN x 2 



-B D ±XS l 



Bu^XS 1 
Avx 2 



BuXS 1 



B D ^XS 

A D x 2 

-B D XS' 



B N =XS 

A N K 2 
—BjyXS' 

Be^XS 1 



\ 
B v XS l 

AuX 2 

\ 
B D XS l 

A D X 2 , 

\ 
BmXS 



A N X 2 



u 

o \ 

B E XS l 



V o o 



-B E XS l A E X j 




AnnX 



(77) 



(78) 



(79) 



(80) 



(81) 



with the desired proportionality condition ([[]) or @ between their diagonal 
and off-diagonal elements. This diagonalization, as one can readily see, is 
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reached by a physically irrelevant rotation (being the same for the up and 
down fermions) with a hierarchically small angle G"^ ~ = = = ~ {jj-) r 
(see®! 

This is, in fact, a crucial point for deriving the LFMG picture within 
the SU(3)f symmetry framework. On the one hand, the scalar rj in the 
Yukawa couplings Wyi (f^2|) must develop VEVs along the first and third 
directions in family space, in order that the proportionality condition (^) 
be satisfied. On the other, these VEVs are allowed to induce at most only 
non-physical extra elements in the mass-matrices M u f and M^f ( |T3"| , [71 ) that 
could be rotated away by some rotation with 0" 3 = Qf 3 . Otherwise, one 
would have large physical mixing of the order of 0(y) (where y « 0.1 is the 
hierarchy parameter introduced in our discussion of neutrino oscillations in 
section 2.2.1) between the first and third families of fermions that is actually 
excluded, at least for quarks where this mixing angle is in fact of order of 
0(y 3 ), as follows from the V u b element value in the observed CKM matrix 
(see®). 

There seems to be no other way of getting the LFMG in the SU(3)p 
model, when the same form Wyi (0) is taken for the Yukawa couplings of 
both the up and down fermions . If so, however, all the matrices M u f, 
and Mtvat appear, as one can readily see from their explicit forms (|77|-|ST|), to 



be largely proportional to each other. This results in an equality of the mass 
ratios for the heavier leptons and quarks 

m c m s m M M N2 M NN2 , . 

■~~ — - ~ ~ (82) 



m t m b m T M N3 M NN3 

which should hold with an accuracy of a few percent (given in fact by the 
ratios of the masses of the first and second families) . While such an equality 
approximately works for the down quarks and charged leptons and also might 
be taken for the Dirac masses of neutrinos, it is certainly unacceptable for the 
up quarks fl6] . This means that we are left with just one possible scenario 
B for the minimal quark mixing in the SU(3)p context, according to which 
the up quarks follow the pattern of direct (1-3) mixing (the alternative II, 
see general discussion in section 2.2), while the down quark mixing is given 
by a mass matrix of the above type (ff8| ) corresponding to the alternative I. 

7 We also neglected the small addition of the order of 0(S 2 ) to the new (33)-elements 
appearing in all these mass-matrices after the diagonalization of the 1-3 blocks. 
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As to the lepton mixing, both of the experimentally allowed scenarios 
A* and B* (section 2.2) follow the alternative I for the Dirac mass-matrices 
of leptons. Thus M N and M E should have the form given above (173, ISO), 



resulting in the proportionality condition ( jB2j) between the Dirac masses of 
neutrinos and charged leptons. This condition should be necessarily kept into 
account when analyzing neutrino oscillations (as we have done in section 
2.2.), since it allows us to predict the hierarchy parameter y appearing in 

the neutrino Dirac mass matrix: y = \ fw^Jm T . If we take (as we actually 
did) that the neutrino Majorana mass matrix also depends on the same 
parameter y then all the characteristic feature of the neutrino oscillations can 
be predicted according to Eq. ( pE2"D in scenario A* and Eq. (^) in scenario 
B*. 

However, scenario A* presupposes the stronger hierarchy fl35p between 
the second and third eigenvalues of the neutrino Majorana mass matrix M/vtv 
to get the observed large mixings (see section 2.2) than the hierarchy (j34[) 
appearing in the Dirac mass matrix of neutrinos. The requirement for 
such a hierarchy to be in the matrix Maw is certainly in conflict with the 
mass proportionality condition (|8"2"D, which appears in scenario A* within the 
SU(3)p framework. Another drawback of this scenario A* in the SU(3)p 
framework is the absence of the antisymmetric mixing term in the Yukawa 
couplings for the right-handed neutrinos (see (0)) that leads to one massless 



Majorana state in M^n (pll). Thus, scenario A* of lepton flavour mixing 
should be dropped in favour of scenario B*, with direct (1-3) mixing in the 
matrix M NN just as it appears for the up quarks in scenario B (see section 
3.3.5 for further discussion of the matrix M NN ). So, we can conclude that 
the combined scenario B+B* is in substance the only one admitted in the 
SU(3)f theory. For this case the mass proportionality condition (p2|) takes 
the reduced form: 

m s m M M N2 . . 

— ~ — - ~ — — (83) 
m b m T M N3 

which seem to work better than the whole equation ([82]), however, not yet 
in a quite satisfactory way. Actually, this relation between the masses of the 
down quarks and leptons is similar to the usual GUT relation, while arising 
in a different way. However, by enlarging the set of horizontal scalars, one 
can rearrange or completely decouple the masses of the quarks and leptons. 
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Now in scenario B, after family symmetry breaking according to the VEVs 
of the horizontal triplets (|BTD, one is immediately led from the Yukawa cou- 
plings collected in Wyi to the mass matrix for the up quarks: 



M u =m° u 



( A vi z 2 A vl zZ - B v Yr 

_^ A U2 Y 2 V i j (84) 

V A m zZ + B v Yl n A m Z 2 



where the mass factor is given by the electroweak symmetry scale involved 

ml =< H u > (85) 

Further, after diagonalization of the symmetric terms in the 1-3 block, one 
comes to the familiar form 



/ -BuYl 



M v = m'jj 



_ A U2 Y 2 I k | (86) 

V B V YF A m Z 2 

discussed in section 2 (see (p2|)). 

Let us note that the above partial diagonalization of the mass matrix ([84]) 
was in fact reached by a rotation for the up quarks with the hierarchically 
small angle 6f 3 ~ — -| = j? ~ (^) r (see (|64]) ). It is crucial for the scenario 
B+B* considered that this rotation turns out to be physically irrelevant: it is 
in fact the same as for the down quarks and leptons, despite the latter having 
mass-matrices of another type ( |74] ) or (|78|). However, the orthogonality (|60|) 
of the VEVs of the scalars involved keeps the rotations in both sectors strictly 
equivalent to one another 

T(« = o - | + 1 = o - e?f E = e£ (87) 

As a result, one can simultaneously transform the primary ( "unrotated" ) 
matrices for the up quarks (|84]) , on the one hand, and those of the down 
quarks ( |T4"D and leptons (neutrinos ( [73] ) and charged leptons |7|)), on the 
other, into the final ( "rotated" ) matrices (|6|) and (|78| - pOD respectively. These 
matrices are just those which exactly correspond to the combined scenario 
B+B* for flavour mixing of quarks and leptons. These matrices follow, as we 
have seen above, from the Yukawa couplings Wyi (|7l|) and Wy2 (^1) when 
the family symmetry SU(3) p spontaneously breaks. 
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3.3.4 Hierarchies in masses and mixings 

We saw in the previous sections that the extra U(l) symmetry, requiring the 
same powers k, n and I of the hierarchy making scalars I and S in similar 
Yukawa couplings of quarks and leptons, leads in the considered scenario 
B+B* to similar hierarchical mass matrices for the down quarks ( |78D and the 
leptons (|7D| , EH). As a result, we obtained the mass relations (B3I) for the 
quarks and leptons in the scenario B+B*. 

However, besides this general conclusion the U(l) symmetry turns out, as 
we now show, to generate the required hierarchy in the mass matrices of the 
quarks and leptons. In order to show this in the most general form, we start 
with a scale-invariant superpotential which, in its only difference with the 
above- used Higgs superpotential W (|56|) , contains no mass parameter for any 
of the scalar superfields, triplets and sextets, involved. Instead, we assume 
that their masses are forbidden by the U(l) symmetry and are generated by 
the VEV of some singlet scalar field T from the following allowed couplings: 



W T = T(A vV rj + Attf + A C C( + A xX x + A^uu) + P(I, S, T) (88) 

So the masses of the scalars, which were directly introduced above (see section 
3.3.1), are now given by 

M Mx *, = A^^- <T> (89) 

through the VEV of the scalar field T and the coupling constants A n ^ x ^. 
Thus, the VEV of the scalar field T gives the basic flavour scale Mp in 
the model, which is proposed now to appear dynamically rather than being 
introduced by hand. This, in substance, should stem from the polynomial 
P(I, S, T) which includes all three singlet scalars; one of which (T) gives 
the basic mass scale in the model, while the other two (J and S) determine 
the mass and mixing hierarchy of the quarks and leptons. The hierarchy 
parameters are basically given by the characteristic VEV ratios I/T and 
S/T determined by the proper minimum of the polynomial P(I,S,T) (see 
below). 

Based on the above comments, we can now write the new total Higgs 



superpotential (see Eqs. (|56|, |57], a| pq) ) in the form 
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W tot = T(A vV fJ + A^ + A c (C + A x xx + A UJ uuJ) + 

+F ■ e a ^e a '^' X{aa , } x_ {m u M} + F ■ e a/3j e a ^x {aa ' } X {m ^ {W} + 
+a ■ r](S r M F + b ■ u((S p (90) 

with I and S standing for I/Mp and S/Mp and the polynomial P(I,S,T) 
being as yet omitted. One can now quickly derive, from the new total Higgs 
superpotential W to t and the Yukawa couplings Wyi (0) and Wyi (|72|), that 
the £7(1) charges of all the horizontal scalar fields involved are given as follows 
in terms of the charges of the singlet scalar s S and /: 

_ 5i_ 8n-2k 
Qr] — ~~^^ s ^ 3 — 

21 5n-2k^ 

Qz = -jQs + — 3 — Qi, 

21 lOn-fc 

Qc = + — g — Qi, 

4Z + 3p Un-5k 
Qx — 3 — Qs H g Qi, 

l + 3p 8n + k 

Qu) — 5 — Qs H t — W) 

3 o 

4-77 — /r; 

Qt = -IQs + ^—Qi (91) 

The U(l) charges for their supersymmetric counterparts fj, £, £, X and To are 
given by 

QrjgZxP = ~Qv,U,X,u + 2( 5r ( 92 ) 

In fact, the U(l) charges of all the horizontal scalars can be expressed in terms 
of one independent charge Qs, as one can readily see from the relation 

Qi = T^tQs (93) 
Sn — k 

arising from the r]( mixing term in the total superpotential Wtot- Here the 
numbers k, I and n are just the powers of the singlet scalars I and S in the 
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Yukawa couplings Wyi and Wy2, while the numbers r and p appear as the 
powers of S in the triplet-triplet and triplet-sextet mixing terms respectively 
in the superpotential W to t ®)- 

When the above horizontal superfields undergo a £7(1) transformation, 
according to the charges (|9i~D, the total superpotential W to t acquires a phys- 
ically inessential overall phased 

W tot -> e l3aT W to t (94) 

where «t is the transformation phase corresponding to the singlet scalar 
superfield T. Under the same transformation, the Yukawa couplings Wyi 
dnj) and W Y2 (0) are left invariant. 

As to the matter superfields, the U(l) charges of the quarks 

, U%, D a R (95) 

a 

leptons 

(e l l ) > x** E ° R (96) 

(see (|50| , |5l"D ) and the ordinary Higgs doublets, H u and H d , are in substance 
arbitrary^ and can be chosen in many possible ways. 

We will now investigate which powers k, I, n and r are required to get 
the observed mass and mixing hierarchy from the quark and lepton mass 
matrices. First of all one should make sure that no other matrix elements are 
allowed, by the U(l) symmetry and the chosen powers, than those required 
by the scenario B+B* in all four mass matrices considered; namely for the up 
quarks (Mu (H)), down quarks (Md ff§|)), neutrinos (M^ (f79|) ) and charged 
leptons (Me (BDH) . One can readily check that for an even value of r and an 
odd value of k with 

n > k (k is odd) (97) 

8 This means that the U(l) considered is in fact the global i?-symmetry U(1)r of the 
superpotential Wtot (|90|)- 

9 One can immediately check that, in the B+ B* scenario, the Yukawa couplings Wyi 
( |7l| ) and Wyi ( |72|) only give five equations for the eight charges (or phases) of the six 
independent fermion |96|) and two Higgs (H u , Hd) multiplets involved. 
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the undesired horizontal scalar combinations of type rji], rj£, r]( and (ac- 
companied by any powers of the singlet scalars / and S) in the Yukawa 
couplings Wyi ([71]), as well as the combinations CCXi ££x and (x i n the 



Yukawa couplings Wy2 (|72|) are strictly prohibited to appear. This is im- 
portant since their presence might crucially destroy scenario B+B*. The 
only allowed extra combination is £x which appears in Wy2, thus giving an 
additional Yukawa coupling 

7? f R%H d [B' df ■ e a ^ p l n ]x Wp} (98) 

This actually turns out to be negligibly small as compared to the other 
couplings for the down quarks and leptons (/ = D,N,E) involved. So, 
one can see that the form of all the mass matrices considered are highly 
protected by the U(l) symmetry from any scalar contributions, other than 
those allowed by the B+B* scenario of the minimal mixing of quarks and 
leptons. 

Let us consider now the mass matrices Mp, and Mp (|78|-|50[). All 



of them have a similar structure with the proportionality (|l|) between their 
diagonal and off-diagonal matrix elements. The natural orders of magnitude 
of these mass matrix elements are given by powers of the VEV of the singlet 
scalar S scaled by the basic mass Mp (y = 



(M f ) 33 pam°, (M f ) 22 ^m°x 2 ^m°y 2r , (M / ) 23 « m°y\ (M/) 12 « m°y l+r 

(99) 

(/ = D, N, E) as directly follows from equations (|78] - |80"1) and (|4]). Since for 
the down quarks the approximate equality (Md) 2 2 ~ (M^)23 (or equivalently 
m s pa y/rnjrfib) is known to work well phenomenologically, one is led to the 
following relation between the powers I and r: 



/ = 2r (100) 

Thus, according to the scenario B+B* considered, all three matrices Md, 
Mn and Me should have a similar hierarchical structure of the above type 
(|99D with I = 2r where r is as yet an arbitrary even number. 

We turn now to the mass matrix My (|86|) for the up quarks. The natural 
orders of magnitude of its elements are given by the appropriate powers of 
the VEV of another singlet I (scaled by Mp). Taking for simplicity yj pa y s 
= y (< I > pa < S >), one readily finds for the matrix elements of Mjj\ 
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(Mi 



U)33 



m 



(Mr 



U)22 



m° uy k , 



(Mi 



U)13 



m°uy n 



(101) 



For consistency with the phenomenological observation [L6| concerning the 
quark masses of the second and third families, (m c /m t ) 2 ~ (m s /mb) 3 , com- 
parison of the diagonal matrix elements of Mu and Mo gives, for the case 
of an even value for r and an odd value of k (see eq. (p7|)), a relation of the 
type 

k = 3r + 1 (102) 

And, finally, according to the inequality fl97|), the minimal choice for the 
power n is 



n 



3r + 2 



(103) 



One can see that the above relations for the powers k, I and n lead to ap- 
proximately the same hierarchy in the mass matrices of quarks and leptons 
for any (even) value of the number r. However, we will proceed further with 
r = 2, thus finally establishing the values 



k 



I 



n 



(104) 



for the other powers. 

According to these values, the quark and lepton mass matrices (|78|-^0|) 
and (|86|) in the scenario B+B* acquire the hierarchical forms: 



M f 



my 



and 



/ 



Xp f y e 



2„,4 



-Xp f y b X 2 y 



-PfV 



PfV 



f — D,N,E 



(105) 



Mr, 



( 



V ay* 




(106) 



Here the parameters Pf , a and a, being ratios of Yukawa coupling constants, 
are all proposed to be of order unity, while the mass scale factors mj and 
mu are as yet "unrotated" masses of the heaviest third family of fermions 
involved. Also, we have parameterized the common VEV ratio = contained 
in the matrices Md,n,e as = = Xy 2 (X ~ 1), according to the behaviour of 
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the small components of the VEVs appearing on the horizontal triplets r\ and 
( (see fl64D). Note that, in the matrices Mf (|105|) , we have not included the 
negligibly small matrix element (Mj) 13 = 0(y 8 ) stemming generally from 
the "extra" Yukawa coupling (|98DH- 

One can see that, due to the high values of the powers in equation ( |104| ), 
even a very smooth starting hierarchy^ y « (m s /m h ) 1 ^ « 0.4 leads, as is 
apparent in the mass matrices (|105|) and (|106|) , to the observed hierarchy in 
the masses and mixings of the quarks and leptons. Apart from the values of 
the weak mixing angles fl27|) which, as was already shown in section 2.1, are 
in a good agreement with experiment, there appear characteristic relations 
between the quark (and lepton) masses of the type: 



m u /m c \ g / 7 m d m s m u /m d \ 9 / 4 m c fm s \ 7 / 4 
m c \m t J ' m s m c \m s J ' m t Vm&/ 



(107) 



which seem to be successfull phenomenologically. Actually, one needs to 
know only the heaviest mass for each family of fermions — the other masses 
are then given in terms of this mass and the hierarchy parameter y. At the 
same time it is well to bear in mind that all these mass relations appear at 
the flavour scale Mp, which could be as high as the GUT scale or even the 
Planck scale. Doing so, one then needs to project these relations down to 
laboratory energies to compare them with experiment [fiOfl . 

We consider finally the form of the polynomial P(I, S, T) introduced 
above (see Eq. (|88|) and how the VEVs of the basic singlet scalars /, S 
and T are determined. Note first of all that it is not immediately evident, 
for the U(l) charges of the scalars J, S and T already determined from 
W to t, Wyi and Wy2 (see Eqs. ( 0T| , and from the hierarchy requirements 
100| , |102| , [103| ), that such a non-trivial U(l) invariant polynomial exists. Re- 



10 This element is, as one can readily see from matrix (105), of order of the lightest mass 
for down quarks. Being inessential in any other respect it will, however, contribute to 
the V u b element, thus changing the formula ( p7| ) given in section 2 to a formula of the type 

\V u b\ — S13 ~ + e* 7 ^ . In the maximal CP violation case (see subsection 3.3.6), 

7 = 5, this new contribution turns out to be about 10 percent of the main one, and is 
neglected below. 

11 To avoid confusion we should note that the hierarchy parameter y introduced here 
differs from that used in section 2.2 (and denoted by the same letter): the old y appears 
to be approximately equal to the square of the new one. 
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markably, however, just for these charges of the three singlet scalars /, S and 
T a polynomial of the type 

P(I,S,T)=a ABC T A I B S c (108) 

actually exists, where the summation over all the allowed integer powers A, 
B and C is imposed^- This polynomial determines the basic mass scale 
Mp ~ < T > in the model, as well as the characteristic hierarchies related 
with the two other VEVs < I > and < S > (all the high-order terms in P 
are scaled by appropriate powers of the mass Mp contained in the coupling 
constants oasc)- 

Actually, in order for a term in this invariant polynomial to exist, the 
U(l) symmetry condition 

AQ T + BQj + CQ S = SQ T (109) 

for the charges of the scalars /, S and T should be satisfied. Substitution 
of the values of these charges, expressed (see (Eqs. |9l], |100| , |102| , |103| )) in 
terms of the power r of the scalar S in the total Higgs superpotential W to t 
(|90D , in this condition gives the equation 

^(7r + 5) + !* + C^ = (110) 

There are only a finite number of positive integer solutions of this equation 
for the powers A, B and C. For the case r = 2 taken in our model, only the 
following four terms appear in P(I, S, T) 



P(I, S, T) = a 300 T 3 + a 241 T 2 I 4 S + a^T'l'S 2 + a 0123 I 12 S 3 (111) 

while not disturbing the other parts of the total Higgs superpotential W to t 
(|9"0"D. In this case one can show from the corresponding F-term equations 
(together with those stemming from Wtot) that in general non-zero VEVs 
appear for all the singlet scalars /, S and T. Even a very smooth hierarchy 
between them could then lead, as was argued above, to the observed hierarchy 
in masses and mixings of the quarks and leptons. 

12 Non-integer or negative values are of course not allowed for the powers in the Higgs 
superpotential or in the Yukawa couplings, due to their generic analyticity. 
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3.3.5 Neutrino masses and oscillations 

We have not yet considered the masses of the right-handed neutrinos and 
the Majorana mass matrix M^n needed to construct the effective mass ma- 
trix M v for the ordinary neutrinos (|3"3"|), using the see-saw mechanism PD| ]. 
According to the B+B* scenario, the right-handed neutrinos should have a 
mass matrix similar to that of the up quarks, having symmetric Yukawa cou- 
plings with the same structure as Wyi (fflD, but the antisymmetric couplings 
are necessarily absent for Majorana neutrinos. As a result, with only the 
symmetric Yukawa couplings contained in Wyi, one is unavoidably led to 
one massless right-handed neutrino. Therefore, some additional symmetric 
coupling is actually needed^. Fortunately such a coupling, generating right- 
handed neutrino masses, automatically appears in Wyi once the triplet-sextet 
coupling term AW 2 (|70| ) is included in the total superpotential W to t ®)- One 
can then immediately confirm that the horizontal sextet u, with the proper 
U(l) charge value given in fl9T|), is allowed to contribute to Wyi together 
with the triplet pairs (( and ££. Thus, the total Yukawa superpotential for 
the right-handed neutrinos reads as 



Wyu = N%N%[A NN0 ■ u {a p } S p T+A NN1 ■ C a Cp + A NN2 ■ € a £ p I k ] (112) 

The first term in Wyii should, of course, also be included in the Yukawa 
couplings Wyi (O) f° r ^ ne U P Q uar ks. However, this term, while being 
crucial for neutrino mass formation, turns out to be inessential for the up 
quarks (for the high value of the power p of the singlet 5* chosen below) . 

After substituting the VEVs of the sextet oj a p (^B]) and the triplets ( and 
£ (|6T| , |64l) into the Yukawa couplings Wyu ( |112j ), one immediately comes to 



the Majorana mass matrix for the right-handed neutrinos: 

47vatiz 2 A NN1 zZ 
M NN = M F | A NN2 Y 2 l k j (113) 

4mizZ A NN1 Z 2 + A NNO uS p T 



13 One might imagine introducing a symmetric mixing term containing the non-diagonal 
horizontal triplet combination 77^ in the Yukawa coupling Wyi (|7l|). However, as we have 
seen in the above, such a term is strictly prohibited by the £7(1) symmetry 
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This matrix can be written in a convenient hierarchical form, like the other 
mass matrices (|105| , |106|) , using the already established hierarchy indices (see 
( |104p ) and keeping the power p as yet undefined: 



2 



M NN — m NN 



( y A y 

W J (114) 

\y 2 1 + PyP 

Here m^N, P and 7 are appropriate combinations of the masses and coupling 
constants in ( |113j ) and the value T =jf- — 1 has been used. The physical 
masses of the Majorana neutrinos are then given by the equations 



l NNl 



m NN (3y 4+p , M NN2 ~ m NN jy 7 , M Nm ~ m NN (115) 



The mass matrix M v for the physical neutrinos can now readily be con- 
structed according to the see-saw formula (|33|), with the neutrino Dirac mass 



matrix (|105|) taken in its "unrotated" form ([73] 

AY \y 



M N = m N \ AV p N y 4 | (116) 
Xy 2 -p N y 4 1 



and the Majorana mass matrix M^n given above (|114|) . The inverse of the 
Majorana mass matrix M^n is readily formed 



2 



1 / l + (3y p -y 
Mim=a-^i PT l y-" +p I , (117) 

and gives rise to an effective light neutrino mass matrix M v . To simplify its 
form we take, as in the general phenomenological case (see section 2.2), for 
all the order-one parameters X, Pn and (3 contained in the matrices and 
M NN , an extra condition of the type (|3~8|): 

|A-l|<y 4 (A = \,p N ,P) (118) 

according to which they are supposed to be equal to unity with a few percent 
accuracy 11 . After this simplification and the choice p — 8, the matrix M v 
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takes a transparent symmetrical form (ignoring some inessential higher order 
terms) 



(119) 



-y 2 -y 2 \ 

M„ = m„ | -y 2 1 + 7?/- 1 1 + 7 ?T 1 <5 

1+75?/- 1 1 +72/^(1 + $ 2 ) J 
where the matrix M v has been first constructed from matrices ( |116| ) and 




( |117| ) according to the see-saw formula ([33]) and then brought to the 
appropriate physical basis, by making a rotation in the (1-3) block (equal to 
that already applied to the matrix of the charged leptons M E so as to bring 
it to the form ( |105| )). Here the mass parameter 



m 



mu = -^r x y (120) 

m NN 

essentially corresponds to the mass of the heaviest neutrino, while the pa- 
rameter S = satisfies \5\ < 1 according to the condition ( |118| ) above. 

One can see that the matrix M v is crucially dependent on the parameter 
7 related to the triplet-triplet term in the Yukawa superpotential Wyn 
( |112| ). While all the dimensionless coupling constants in Wyn are proposed 
(just as for the other Yukawa and Higgs coupling constants in the model) to 
be of the natural order 0(1), the parameter 7 should be considerably smaller 
to have an experimentally acceptable mass interval between the masses of the 
second and third family neutrinos. In particular, if we take 7 ~ y A (which is 
just of the order 0(m s /mb), see the matrices (|105|) ), we come to an attractive 
picture for neutrino masses and oscillations. Actually, after diagonalization 
of the matrix M u , one can readily see that the ratios of the neutrino masses 
are0 

1 x 

m ul : m u2 : m u3 = -y 7 : -y 3 : 1 (121) 

and the small-mixing angle MSW solution (SMA) for neutrino oscillations 
naturally emerges, with the following predictions for the basic parameters: 



sin 2 2£ aim ^l, sm 2 26 sun <2y\ U e3 ~ ~ £ (122) 

V2 A<» 16 



14 

are a little changed, m v \ : m v i : m V 3 = jy 7 : ^y 3 : 1 



For simplicity, we take S = 1 here. For the other limiting case, 5 = 0, the mass ratios 
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Here, in deriving the limit on the contribution to sin 2 29 sun from the neu- 
tral lepton sector, we have used the inequality (|118|) for the (M„)i 2 element 
obtained after successive (2-3) and (1-3) block diagonalizations. 

For the phenomenological value of the hierarchy parameter in our model, 
y ps (m s /m fc ) 1//4 ps 0.4, the predictions (|122|) turn out to be inside of the 
experimentally allowed intervals for the SMA solution (JE|). Note that for 
the case when the above parameters X, Pn and (3 are strictly equal to unity 
the (1-2) mixing completely disappears, while the (2-3) mixing determining 
the value of sin 2 20 atm is left maximal. Note also that in the SMA case one 
must expect, as we mentioned above (see section 2.2), a sizable contribu- 
tion to sin 2 26' sun stemming from the charged lepton sector (see Eqs.flSlp). 
Remarkably, this contribution taken alone 

sin 2 29 sun ~ 4 sin 2 9 efl = 4^ (123) 

gives a good approximation to the SMA solution. 

So, one can conclude that the SU (3) p theory of flavour successfully de- 
scribes the neutrino masses and mixings as well. Remarkably, just the neu- 
trino oscillation phenomena already observed could give the first real hint 
as to where the scale Mp of the SU(3)p flavour symmetry might be lo- 
cated. Actually, if the right-handed neutrinos are not sterile under SU(3)f 
(as they are under all other gauge symmetries) then their masses should 
be directly related to the flavour scale, just like the masses of the quarks 
and leptons are related to the electroweak scale. We treat the right-handed 
neutrinos, like the other quarks and leptons, as triplets under SU(3)f 
(see (|5T|)) which acquire their masses from the SU(3)f invariant couplings 
Wyu ( |112[ ), when the horizontal scalars develop VEVs and flavour symme- 
try breaks. Due to the chiral nature of SU(3)p, there appear no mass terms 
for the Majorana neutrinos when the symmetry is unbroken. Thus, it is 
apparent that the mass of the heaviest Majorana neutrino, appearing from 
the Yukawa coupling Wyu, should be of the same order as the flavour scale 
M F , since the basic Yukawa coupling constants are all proposed to be of or- 
der unity. Using now equation (|120|) , with the heaviest Dirac neutrino mass 
rrijv taken as large as the top quark mass m t , the heaviest physical neutrino 



mass m u determined from the experimentally observed value of Am„ im [IS 



m v ~ y Am 2 tm ~ 0.07 eV and the hierarchy parameter y ps (m s /mi,) 1 ' /4 , one 
comes to an approximate value for the flavour scale 
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A' 



m, 



-7 1/ 



m, 



r^/^3/^ iol6 ^ 



mj 



(124) 



from which the masses and mixings of the quarks and leptons are basically 
generated. Due to the enhancement factor •y~ 1 y appearing in the SU(3)p 
model considered, this scale turns out to be as large as the well-known SUSY 
GUT scale [4~I|. The high scale Mp of the flavour symmetry found above (see 



(|124 )) would mean that the SU(3)f flavour symmetry could be part of some 
extended SUSY GUT, either in a direct product form like SU(5) (g> SU(3)f 
[|], 50(10) ® SU(3) F || and E(6) ® SU(3) F @ or even in the form of a 
family unifying simple group like SU(8) GUT |35[| . 



3.3.6 Spontaneous CP violation 

It seems that the whole picture of flavour mixing of quarks and leptons, 
including the hierarchical pattern of the masses and mixing angles, can arise 
as a consequence of the spontaneous breakdown of the family symmetry 
SU(3)f considered. We show, in this section, that even CP violation can be 
spontaneously induced in the SU(3)f theory of flavour. Furthermore, when 
it occurs, one is unavoidably led to maximal CP violation. 

The essential point is that, in general, the horizontal scalars (|5f|) develop 
complex VEVs (|6"1| , |66| ) leading to CP violation, even if all the coupling 
constants in the superpotential W to t ( pOf) and the Yukawa couplings in Wyi 
( |7T|) and Wy2 ( 1^2]) are taken to be real. Let us consider this in more detail. 
For simplicity, one can always choose real VEVs for the singlet scalars /, 
S and T, among the possible VEV solutions stemming from the invariant 
polynomial P(J, S, T) ( |108| ). Then, as follows from the triplet VEV equations 
( [59] , |H), the phases of the large and small components (|6l|) of their VEVs 
should satisfy the conditions 



Sx + Sx = 0, 5 Y + 5 Y = 0, 5 Z + % = 0, 5 X + S Y + S z = n (125) 
dx = 7T - 5 X + S Y + 25 z , S z — 7T — 5x + 25 z , S-g = n - 5 X 

There is one more non-trivial condition, stemming from the imaginary part of 
the triplet F-term equations (|58D containing also the singlet scalar S. When 
one uses the previous conditions ( |125| ), this extra condition reads as follows: 
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7T 



0,1 



(126) 



Note now that, due to the exact U(l) symmetry of the total superpotential 
W to t (TO and the polynomial P(I,S,T) ( |1(J8[ ), one can always choose the 
phase of the U (1) transformation in such a way as to make one of the phases 
5x, Sy or 5z in the equations (|125| , |126[) vanish. Thus, taking 5y = 0, one 
finally comes to two non-trivial solutions for the phases of the large and small 
components of the triplet VEVs: 



5 X = vr/4, 5 Z = 3vr/4, 

= tt/4, 4 = vr/4, 5- = 3tt/4 (127) 



and 



6 X = tt/2, 6 z = n/2, 

6s = 3vr/2, S z = 3tt/2, 5- = tt/2 (128) 

depending on the two possible values of k, k = and k = 1 respectively. We 
show below by applying these triplet VEV phase values to the quark and lep- 
ton mass matrices that, while the second solution ( |128| ) is CP-conserving, the 
first one ( |127|) leads to maximal CP-violation. There are no other solutions 
for the phases of the triplet VEVs in the SU(3)f model considered^]. 

Let us consider the CP-conserving case ( |128j ) first. As one can quickly 
confirm, the mass matrices M\j ( j36|) and Mo (|78|) of the up and down quarks, 
in this case containing the phases of the horizontal triplet VEVs given by 
equation (|128|) , are strictly Hermitian with real diagonal elements and pure 
imaginary off-diagonal ones. Such matrices Mu and Mp, with five and three 
texture zero structure respectively, cannot lead to CP violation — the phase 
of the corresponding Jarlskog determinant El vanishes identically. 



By contrast, the CP- violating case (|127|) results in non-Hermitian mass 
matrices Mjj and Mo- One can find their explicit form, in terms of the masses 



15 As to the VEVs of the horizontal sextets x an d w they always can be chosen real. 
Actually, by a proper choice of the phase of the SU(3)f transformations (particularly its 
A 8 subgroup transformation) one can take the phase of < x > to be zero. Then, as follows 
from the sextet F-term equations (|65|), the phase of < u> > turns out to vanish as well. 
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of the up and down quarks and the triplet VEV phases ( 127|) , by forming their 



Hermitian products MjjMfj and MpM^ and then using the conservation of 
their traces, determinants and sums of principal minors. Doing so, we find 
for the matrices Mu and Mu>: 

/ -^/m u m t e in/2 

Mu = m c e™' 2 | (129) 

\ v /m u m t e l7r /' 2 m t — m u 

and 



M, 



D 



I -^?^W s e- i3n ^ \ 

y / m^mje~ ;i37r / 4 m s v /m ( jm(,e ,3 ' r ^ 4 

\ "V^Se*'' 4 m b — TTT'd J 



(130) 



where we have removed the inessential common phase factors. 

Notably, while the diagonalization of the non-Hermitian matrices Mu 
and Md requires different unitary transformations for the left-handed and 
right-handed quarks, the Hermitian constructions \I( \I r and M^M^ are di- 
agonalized by unitary transformations acting on just the left-handed quarks. 
These are precisely the transformations which contribute to the weak inter- 
action mixings of quarks collected in the CKM matrix. Thus, one must di- 
agonalize the MuMy and MpM^ first and then construct the CKM matrix. 
Proceeding in such a manner, we immediately find that the diagonalization 
of \ li \ I r and MdM^ is carried out by unitary matrices of the type 



V, 



u 



D 



u 



(131) 



They in fact contain the same plane rotations K[ 3 (s 



13 



m u /mt), R?2 



(s 12 — \jm d /m s ) and R^ 3 (s 2 3 — ymd/rrib) as those used for the diagonal- 
ization of the phenomenological Hermitian mass matrices Mu (E^) and Mjj 
(HD considered in section 2.2. However, the phase matrices <& u and $ D now 
become 



u 








D 



1 
e 




-i37r/4 



(132) 
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Finally, substituting all the above rotations and phases into the CKM matrix 
Vckm = VuVp (|14]) and properly rephasing the c quark field (c — > e i37r//4 c), 



one comes to 



CKM 



1 C12C13 - is 12 Si 3 S 2 3 -S12C13 - ^i 3 S 2 3Ci2 -iSi 3 C 2 3 

S12C23 C12C23 -s 2 3 J (133) 

V -S13C12 - 2S12S23C13 S13S12 - «S 2 3Ci2Ci3 -iCi 3 C 23 



While this is in substance a new parameterization for the CKM matrix, it 



is quite close to the standard parameterization |]Ib[ , as one can show by 
comparing the moduli of the matrix elements in them both. In such a way, 
one readily concludes that the CP violation phase in Vckm (|133|) is in fact 
maximal, 5 = n/2, while all the mixing angles are rather small being basically 
determined by the masses of the lightest quarks u and d (see Eqs. (p7[)), as 
was expected. 

4 Conclusions 

The present observational status of quark flavour mixing, as described by the 
CKM matrix elements flEfl , shows that the third family t and b quarks are 
largely decoupled from the lighter families. At first sight, it looks quite sur- 
prising that not only the 1-3 "far neighbour" mixing (giving the V u b element 
in Vckm) but also the 2-3 "nearest neighbour" mixing (V c b) happen to be 
small compared with the "ordinary" 1-2 Cabibbo mixing (V us ) which is deter- 
mined, according to common belief, by the lightest u and d quarks. This led 
us to the idea that all the other mixings, and primarily the 2-3 mixing, could 
also be controlled by the masses m u and rrid and that the above-mentioned 
decoupling of the third family t and b quarks is determined by the square 
roots of the corresponding mass ratios y ^f- and respectively. So, in 
the chiral symmetry limit m u = = 0, not only does CP violation vanish, 
as argued in but all the flavour mixings disappear as well. 

In such a way the Lightest Family Mass Generation (LFMG) mechanism 
for flavour mixing of quarks was formulated in section 2.1, with two possible 
scenarios A and B. We found that the LFMG mechanism reproduces well 
the values of the already well measured CKM matrix elements and gives dis- 
tinctive predictions for the yet poorly known ones, in both scenarios A and 
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B. One could say that, for the first time, there are compact working formu- 
lae (especially compact in scenario B) for all the CKM angles in terms of 
quark mass ratios. The only unknown parameter is the CP violating phase 
5. Taking it to be maximal (5 = |), we obtain a full determination of the 
CKM matrix that is numerically summarized in section 2.1. The LFMG 
mechanism was extended to the lepton sector in section 2.2, again with two 
possible scenarios A* and B*. These were shown to be quite successful phe- 
nomenologically as well, naturally leading to a consistent description of the 
presently observed situation in solar and atmospheric neutrino oscillations . 
Remarkably, the same mechanism results simultaneously in small quark and 
large lepton mixing. 

From the theoretical point of view, the basic LFMG mechanism arises 
from the generic proportionality condition (|]) between diagonal and off- 
diagonal elements of the mass matrices. One might think that this con- 
dition suggests some underlying flavour symmetry, probably a non-abelian 
SU (N) symmetry, treating the N families in a special way. Indeed, for N = 3 
families, we have found that the local SU(3)f chiral family (or horizontal) 
symmetry ||, considered in detail in section 3, seems to be a good candidate. 
The SU(3)f family symmetry is accompanied by an additional global U(l) 
symmetry, which seems to be very important for the recovery of the final 
picture of flavour mixing of quarks and leptons. There is a whole plethora 
of SU(3)p invariant Yukawa couplings available. However, due to the extra 
protecting symmetry £7(1), one is able to choose them in a special form which 
leads to the minimal flavour mixing of quarks and leptons. There are four 
pairs of experimentally allowed scenarios (see section 2) for the combined 
quark and lepton flavour mixings: A+A*, A+ B*, B+A* and/or B+B*. We 
found that the SU(3)f theory only admits the scenario B+B* which, for- 
tunately, seems to provide the best fit to the experimental data on quark 
mixing. It also leads to the SMA solution for the solar neutrino problem. 
Besides its crucial role in selecting the pattern of flavour mixing among the 
four scenarios involved, the U(l) symmetry turns out, as we have shown, 
to determine the form of the hierarchy in the quark and lepton mass matri- 
ces^]. Actually, one only needs to know the heaviest mass for each family 

16 Note, however, that we had to take the parameter 7 in the Majorana mass matrix (|114|) 
to be small (7 ~ y 4 ), while all the parameters in the quark and lepton mass matrices are 
supposed to of the order 0(1). 
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of fermions — the other masses are then given in terms of this mass and the 
input hierarchy parameter y. 

The high scale Mp of the flavour symmetry found here from neutrino 
oscillations means that the local flavour symmetry SU(3)p might be inter- 
preted as part of some extended SUSY GUT; this could appear in a direct 
product form like SU(5) ® SU(3) F , 50(10) ® SU(3) F and E(Q) ® SU(3) F or 
even in the form of a simple group like the 577(8) GUT unifying all families. 

And the last point, which is especially noteworthy, is that the symmetry- 
breaking horizontal scalar fields, triplets and sextets of 577(3), in general 
develop complex VEVs and, in cases linked to the LFMG mechanism, trans- 
mit a maximal CP violating phase 5 = | to the effective Yukawa couplings 
involved. Apart from the direct predictability of 5, the possibility that CP 
symmetry is broken spontaneously like other fundamental symmetries of the 
Standard Model seems very attractive — both aesthetically and because it 
gives some clue to the origin of maximal CP violation in the CKM matrix. 

So, an S77(3) family symmetry seems to be a good candidate for the 
basic theory underlying our proposed LFMG mechanism, although we do 
not exclude the possibility of other interpretations as well. Certainly, even 
without a theoretical derivation of Eq. ([[]), the LFMG mechanism can be 
considered as a successful predictive ansatz in its own right. Its further 
testing could shed light on the underlying flavour dynamics and the way 
towards the final theory of flavour. 
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